Use of Visual Information in Experimental End-of-Speech Detection

Technical use of visual information in human machine dialog management presupposes a use of visual cues by humans. The current study tries to determine to what extent human subjects can use visual cues to detect Transition Relevance Places (TRPs) in prerecorded and annotated continuous dialogs. Subjects were asked to utter minimal responses to dialog recordings that were presented Audio-Visually, as well as Audio-only and Visual-only. Their responses were compared to turn switch delays in the original dialogs. Although there is unambiguous evidence that visual cues alone can guide TRP detection, the task proved to be difficult. An estimated three out of four responses were spurious. Audio-only presentation was felt to be easier, but the results of the full Audio-Visual presentation were more accurate. Some, as yet inconclusive, evidence was found for a timing effect of a gaze direction switch towards the listener.